Papers with frontier LMs
Can Language Models Reason about Individualistic Human Values and Preferences? (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods and evaluation frameworks for achieving pluralistic alignment are limited by the diversity of people, which is pre-specified and coarsely categorized, papering over individuality. |
| Approach: | They propose to use a dataset transformed from the influential World Values Survey to study language models on the specific challenge of individualistic value reasoning. |
| Outcome: | The proposed model can predict individualistic values with accuracies between 55% and 65%, while a precise description of individualistic value judgments cannot be approximated only via demographic information. |
CulturalBench: A Robust, Diverse and Challenging Benchmark for Measuring LMs’ Cultural Knowledge Through Human-AI Red-Teaming (2025.acl-long)
Copied to clipboard
Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz, Yejin Choi
| Challenge: | CulturalBench is a set of 1,696 human-written and human-verified questions to assess LMs’ cultural knowledge covering 45 global regions including underrepresented ones like Bangladesh, Zimbabwe, and Peru. |
| Approach: | They construct a set of 1,696 human-written and human-verified questions to assess LMs' cultural knowledge, covering 45 global regions including underrepresented ones like Bangladesh, Zimbabwe, and Peru. |
| Outcome: | The proposed model outperforms other models across cultures, while underperforming on questions related to North Africa, South America and Middle East. |